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Abstract 

For a tree Markov random field non-reconstruction is said to hold if as the depth of the tree goes to 
infinity the information that a typical configuration at the leaves gives about the value at the root goes 
to zero. The distribution of the measure at the root conditioned on a typical boundary can be computed 
using a distributional recurrence. However the exact computation is not feasible because the support of 
the distribution grows exponentially with the depth. 

In this work, we introduce a notion of a survey of a distribution over probability vectors which is a 
succinct representation of the true distribution. We show that a survey of the distribution of the measure 
at the root can be constructed by an efficient recursive algorithm. The key properties of surveys are 
that the size does not grow with the depth, they can be constructed recursively, and they still provide a 
good bound for the distance between the true conditional distribution and the unconditional distribution 
at the root. This approach applies to a large class of Markov random field models including randomly 
generated ones. As an application we show bounds on the reconstruction threshold for the Potts model 
on small-degree trees. 
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1 Introduction 



Correlations between distant elements or sets of elements in randomly generated Markov random fields 
(MRFs) are a main consideration in the analysis of random constraint satisfaction problems, and in the 
design and analysis of message-passing, local search, and other iterative algorithms for these problems. Here 
we study one such concept of correlation on tree MRFs. The presence of this correlation is known as the 
property of reconstruction, and its absence as non-reconstruction. Non-reconstruction is equivalent to the 
free-boundary Gibbs measure of the tree being extremal [5]. From the point of view of statistical physics, 
reconstruction is equivalent to replica-symmetry breaking (9]]. Some important results on reconstruction 
are j6l[3l[IIl[l2l|2l[T][T5l[T6l. For the connection between this concept and the design of algorithms for 
constraint satisfaction and optimization problems we refer the reader to liTOl [171 181 and the references therein. 

Our contribution is the first general efficient computational method to obtain non-trivial bounds for the 
probability of reconstruction for a tree Markov random field. We illustrate the method with an application 
to the Potts model. 

Consider a tree MRF The distribution of interest is the conditional distribution at the root, given a boundary 
that is generated randomly according to the MRF. It is said that the root cannot be reconstructed, or the 
non-reconstruction property holds, if and only if with high probability this distribution converges to the 
unconditional marginal distribution at the root as the depth of the tree goes to infinity. The distribution of 
the conditional distribution at the root (conditioned on a random boundary) can be expressed recursively - 
there is a simple analytic expression for the distribution for a tree of depth n + 1 in terms of the distribution 
for the tree of depth n. However as the depth of the tree increases the support of the distribution grows 
exponentially, which makes numerical estimates difficult to obtain. For analytical analysis one essentially 
needs to find a special parameter of the distribution which can be bounded recursively, and this depends 
on the particular MRF. Often the analysis has two steps - first showing that the expected distance from the 
unconditional distribution is below some small constant, and then showing that arriving below this small 
constant implies that this distance (or some parameter related to it) decreases geometrically. Particular 
examples of this approach are the recent results on reconstruction for colorings of Bhatnagar, Vera, Vigoda, 
and Weitz HI and Sly and for the q-state Potts model of Sly lfl6l . The moral from their analysis 
is that the first step is the more difficult to achieve. In particular, it is the step that makes the analysis 
possible only for the case of large-degree trees. Here we propose a very general method for making this first 
step, independent of the particular method for generating the MRF, and which is practical for the case of 
small-degree trees. 

We introduce the notion of a survey of a distribution over probability vectors. The survey can be thought of 
as a "projection" onto a small "basis" set of probability vectors that carries enough information about the 
true distribution. In particular, for the reconstruction problem, when the goal is to bound from above the 
probability of reconstruction, we show that it suffices to keep a survey of the distribution at each iteration. 
Applying the recursion to the surveys for enough iterations allows us to obtain very good (possibly arbitrarily 
good) bounds on the probability of reconstruction. 

We apply our method to the symmetric 3-state Potts model with various parameters in order to compare with 
previous results, although it will be clear that the method does not depend on the symmetry of the model. 

'Notice that the "distribution of ... distribution" is not a mistake - the object we are interested in is indeed the distribution of the 
randomness left in the root after looking at the boundary. 
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For the Potts model the second step is also easy to achieve using the ideas from recent work of Sly |[T6l . 
Since the complexity of the method is exponential in the degree of the tree, and in the alphabet of the MRF, 
we do not apply it to very large parameters. However, we are able to demonstrate new bounds for 3-spins on 
the 2-ary and 3-ary tree, and on a random tree in which every internal vertex has either 2 or 3 children with 
equal probability, improving the bounds of Mossel and Peres lfl2l . We should point out that very recently 
Formentin and Kiilske HI demonstrated even better results for this model. Thus, rather than the numerical 
results, the importance of our contribution is in the generality of our method. 

The algorithm we present can be viewed as a rigorous alternative to the population dynamics algorithm used 
in physics (9j [171 to determine the spin-glass transition. In population dynamics the distributional recursion 
is approximated by keeping a sample of the distribution at each step. What is not known rigorously is 
whether applying the recursion to a sample of the distribution for the tree of depth n really results in some 
sense in a "faithful" sample of the distribution for the tree of depth n + 1. In contrast, the main technical 
lemma of our work is precisely the statement that applying the recursion to the survey of the distribution for 
the tree of depth n results in a survey of the distribution for the tree of depth n + 1. 

Another related algorithm is the density evolution algorithm for analysis of the probability of bit-error of 
LDPC codes lfT3~l[T4l . The density evolution algorithm, as its name indicates, is also a recursion on distribu- 
tions, and in practice it is carried out by heuristically quantizing the distribution at every step by rounding. 
Unfortunately, unlike the reconstruction recursion, the density evolution recursion does not commute with 
taking surveys of the distributions therefore our method cannot be applied, or at least not in the obvious way. 

Quantization of distributions is also an important design step in the Survey Propagation algorithm of |[T0l . In 
its analytical form Survey Propagation uses messages that are distributions with growing support, whereas 
in its practical form the messages are distributions with support of size 3. However we do not know if there 
is a precise mathematical connection between the two kinds of quantization. 

This article is organized as follows. In the next section we give the technical definitions related to recon- 
struction, and the recursion relation that we analyze. Section [3]is dedicated to the new definition of a survey 
of a distributions, its key properties, and how it can be applied to the reconstruction problem. In the last 
section we discuss an application to the Potts model, and compare the results obtained by our method to 
previous bounds. 

2 Reconstruction on trees 

The reconstruction problem in its simplest form can be stated in terms of a broadcast problem. Consider 
a process in which information is broadcast from the root of an infinite rooted tree T to other vertices as 
follows. Each edge e = (u, v) acts as a channel M with a finite alphabet D = {1, 2, . . . , q}. The channel 
M is a Markov chain where (M)ij = Pt(v = j\u = i). 

The letter at the root p, denoted by a p , is chosen according to some initial distribution. This value is then 
propagated in the tree as follows. For vertex v with parent u, let a v = M(a u ) for each edge independently. 

For distributions n and v on the same space f}, the total variation distance between fi and v is 
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Let L(n) denote the configuration at level n. 

Definition 1 The tree T and channel M have the non-reconstruction property if for every a,b £ D, 

lim d T v[L a {n),L b {n)} = 

n— >oo 

where L c (n) denotes the conditional distribution of L(n) given that a p = a 

In 2006 Mezard and Montanari [9 ] showed that the same problem had also been studied in physics for even 
more general models, and the reconstruction transition is the equivalent to the replica symmetry breaking 
spin-glass transition. In the physics formalism rather than using channels, the system is described as a 
Markov random field, which in many application is also randomly generated, and the question again is 
whether the root is reconstructible from typical assignments at the leafs. 

Here, for greatest generality we will define the reconstruction problem using a Markov random field (MRF). 
This way we capture models such as 3-SAT and other constraint satisfaction problems, which are not as easy 
to think about in terms of channels. 

Consider a tree T = (V, E) with root vertex p. For every v € V there is a domain of values D v that 
can be assigned to this vertex, and for every edge e = (u, v) a non-negative function we call potential 
: D u x D v — > R + . We will define a distribution over assignments to the vertices x ve yD v = D. 
For every configuration a G D, let a v denote the component corresponding to vertex v, and a(n) the 
components corresponding to level n of the tree. 

P T ,*(cf) := — ^- tt e (<r„,<7„), (1) 

' e=(u,v)(zE 

where Z is the normalizing constant Z T ^ = Y^weD Tle=(u,v)eE ^e(cr u , a v ). 

We will allow random MRFs (T, ^) generated in the following way. To each level there corresponds a 
degree distribution and a distribution over potential functions. A tree of depth 1 is just a single vertex (no 
potential functions). An MRF of depth n + 1 is generated by choosing a degree for the root from the 
degree distribution of level n + 1, next choosing potential functions for all the edges adjacent to the root 
independently from the distribution of potential functions corresponding to level n + 1, and finally attaching 
randomly generated MRFs of depth n to the other ends of the edges. 

We denote by L{n) the configuration at the vertices at level n (assuming T is of depth at least ri), and by p 
the root. Let also L a (n) denote the configuration at level n conditional on the root having value a £ D p . 

Definition 2 We say that a random MRF has the non-reconstruction property if for every a, 6 € D p 

lim E T ^[d TV (L a (n),L b (n))} =0. 

n— >oo ' 

For the rest of this section we will consider a fixed MRF, so we can omit ^ from the subscript as the functions 
on the edges will not be changing, and we will also omit T from the subscript whenever it is understood. 
We will also use a and b for the events that the root takes the value a or b respectively. 
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We will use an alternative expression for the total variation distance, which follows from B ayes' rule. For 
this we denote ir(a) := F T (a) and ir(b) := F T (b). 

d TV {L a (n),L b (n)) = ^ |P(£(n) = L\a) - P(X(n) = L\b)\ 

L 

^ F(a\L(n) = L)P(X(n) = L) F(b\L(n) = L)F{L(n) = L) 



E 



L(n) 



vr(a) 

F(a\L(n)) F(b\L(nj) 



7T(6) 



(2) 
(3) 



7T(o) 



7t(6) 



(4) 



< 



^yE L(n) [|P(o|L(n)) - 7r(a)|] + ^^L(n) [\nb\L(n)) - tt(6)|] . (5) 



An immediate observation is that this quantity is monotonically decreasing with the depth of the tree n. 

'(a\L(n + l)) F(b\L(n + l)) 



E 



L(n+1) 
E 



L(n+1) 



n(a) 7r(6) 

a|L(n) = L) P(L(n) = L\L(n + 1)) P(6jL(n) = L) P(L(n) = L\L(n + 1)) 



< E 



L(n+l) 



5^P(L(n) = L|L(n + 1)) 



7T(6) 



(a|L(n) = L) P(6|L(n) = L) 



7r(a) 



P(L(n + 1) = L') ^ F(L(n) = L\L(n + 1) = L') 



Tr(fe) 



P(a|L(n) = L) F(b\L{n) = L) 



7r(a) 



tt(6) 



^P(L(n)=L) 



(a|L(n) = L) P(6|L(n) = L) 



E 



L(n) 



7T(o) 

P(a|L(n)) P(6|L(n)) 



7T(6) 



7T(o) 



tt(6) 



This expression for the variation distance is a function of a distribution that we will refer to many times in 
the rest of the paper, so for convenience we define the following terminology for it: 

Definition 3 For a tree Markov random model as in equation £[]) the residual distribution at the root of 
T is the distribution of the marginal distribution at the root conditional on a random boundary, which is 
chosen according to the distribution Ft In other words it is the distribution of the probability vector 
r] = {r] a := Pr(a|L(n)), a G Dp), where L(n) is a configuration for the leaves chosen according to Pt,*- 

Intuitively, a random MRF has the non-reconstruction property if the residual distribution at the root is 
with high probability concentrated on probability vectors arbitrarily close to the stationary distribution tt. 
In particular, we will aim to show that E[|P(o|L(n)) — 7r(a)|] goes to for every a € D p , which implies 
non-reconstruction by the inequality (f5]). 

Next we derive the recursive equation for the residual distribution at the root of a tree. For a tree T of depth 
n with edges Et let 

Z T (a) := ^ e {a u ,a v ). 
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Thus Zt := Ylw Zt{&)- For a boundary configuration L, let 

Z T (L):= £ Z t(°)- 

a:a(n)=L 

For a boundary L let rfr(L) = (rjj,(L) : a £ L> p ) denote the probability vector for the distribution of the 
assignment at the root conditional on the boundary being L, i.e. rj^(L) := Py(a|L(n) = L). For any 
probability vector r\ on D p let 

z T (n) -.= Zt ^- 

L-Vt{L)=v 

Thus the probability that the marginal distribution at the root is equal to rj is Zt{t])/Zt- 

Suppose the children of the root of T are v± , . . . , v r , the corresponding edges connecting them to the root are 
e±, . . . ,e r and the subtrees rooted at them are respectively T\,...,T r . Consider a boundary configuration 
L = (L\, . . . ,L r ) for the large tree T, where Lj is the part of the boundary that belongs to Tj. It is 
straightforward to derive the expression for ry^(L) in terms of rfr^Li): 



Vt(L) 



J2a:a(n)=L,cr =a Z ( a ) 



Z T (L) 



Z T (L) 

It is convenient to define the corresponding function (which is actually the update function of the belief 
propagation algorithm for computing the marginal distribution at the root of an acyclic MRF). Let V v be 
the space of probabilities over the domain D v for any v G V. We define / : V Vl , . . . , V Vr — > V p in the 
following way: 

r 

We will also need to use the norm \\f{r]i, . . . , rj r )\\ = Yl a eD p f a {vii ■ ■ ■ iVr)- With this notation 

f a (r lTl (L 1 ),...,r lTr (L r )) 



\\f(r] Tl (Li),...,r] Tr (L r 



For two vectors the symmetric relation oc will indicate that the vectors are equal up to a multiplicative 
constant. Thus t]t{L) oc / (777^ (Li), . . . , rfr r {L r )). When used with probabilities it will indicate that 
the normalization constant has been omitted. We are now ready to derive the recursion for the residual 
distribution at the root of the tree. 



Theorem 4 Let Pi and Q be random vectors such that Pj is distributed according to the residual distribu- 
tion at the root ofTifor i = 1, . . . , r and Q is distributed according to the residual distribution at the root 
ofT. Then for any r] € V p 

P(Q = r/) ocE[||/(P 1 ,...,P r )|| xInd[/(Pi,...,P r ) ex??]]. 
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Proof: First we derive the recursion for the total weight of configurations with a fixed boundary: 

r 

Z t(L) = E E W^eAa,<Tv t )Z Ti {L t )^{L t ) 

adDp <r vl ,...,<T Vr i=l 

= x ^ fi e ^(l^^k,^) 

\i=l / aeDp i=l(T„ i eD„ i 

= (jlZr^L^ y.\\f{i lTl {L l ),...,r ]Tr {L r ))\\. 

Next, using the above, we derive the recursion for the total weight of configurations yielding a given marginal 
distribution at the root: 

z T ( V ) = e Z ^ L ) 

L:r) T (L)=n 



E \X\ZtXU)) x\\f( VTl (L 1 ),...,r lTr (L r 



L = (L 1 ,...,L r ): \i=l / 

r 

E ii/fai.-»*oii E ••• E n^te) 

r 

e n/( ? ?i'---' r /'-)iin z ^(^)- 

T71,--,l7r: i = l 

Finally we can derive the recursion for the residual distribution at the root: 

P(Q-,) = ^2> 

= S ll/«n,...,, r )llri%^ 

1 VI Vr'- i = l 1 i 

= Hk^i E [||/(P 1 ,...,P r )||xInd[/(P 1 ,...,P r )ar ? ]]. 

■ 

This recursion was also derived by Mezard and Montanari in (9l and, as pointed out by them, its fixed point 
is known as the "1-step Replica Symmetric Breaking solution with Parisi parameter m = 1" (in the general 
1RSB scheme the factor ||/(. . . ) |[ is raised to a power m). Almost all other work in reconstruction considers, 
instead, the recursion for the conditional residual distribution at the root, conditioned on the boundary being 
generated from the MRF with a fixed value at the root. The method we present here can be applied only 
with the unconditional distribution. 

The main contribution of this article is a method for discretizing the recursion of Theorem 01 The only 
property of / that is used is that it is a multi-affine function (i.e. affine in each coordinate). Thus the main 
technical theorem will not use the definitions related to reconstruction. 
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3 Surveys of distributions 



Let us first illustrate our approach with the example of the channel corresponding to random coloring on the 
regular d-aiy tree. Each vertex can take one of q colors. Based on the color of the parent, the colors of the 
children are chosen independently and uniformly at random from the set of colors different from that of the 
parent. If we start with a uniformly random color at the root, this process generates a random coloring of the 
tree. Showing non-reconstruction for this process is equivalent to showing that for almost all colorings of 
the leafs generated by choosing a random coloring of the tree, there are almost the same number of colorings 
consistent with these leafs, in which the root of the tree has each of the q colors. More precisely, let Cl 
denote the colorings of a tree of depth n with leafs colored according to L, and C L denote the colorings in Cl 
with color i at the root. We will recursively try to match up the colorings in the sets C\, by splitting 
them into sets that are balanced or close to balanced. For example, suppose q = 3, the colors are called Red, 
Blue and Green, and for some L we have (C^ , C^ , C^) = (10, 5, 7). Then we can split this set of colorings 
into 3 sets : one that is perfectly balanced, containing 5 colorings of each type; one that is balanced with 
respect to Red and Green, containing two colorings with Red at the root, and two with Green at the root; and 
one that contains the remaining 3 colorings with Red at the root ((10, 5, 7) = (5, 5, 5) + (2, 0, 2) + (3, 0, 0)). 
We call such sets bundles. A bundle always contains only colorings that have the same colors at the leafs. 
Bundles can be of several different types according to the ratio of number of colorings with red, blue and 
green at the root, and we can choose the types that will be allowed. We will only keep track of the number 
of bundles of each type. The goal is to construct the bundles recursively in such a way that the majority of 
colorings are eventually in balanced bundles. 

There is a simple process to construct bundles recursively. Suppose we have a particular splitting of the 
colorings of the d-aiy tree of depth n into bundles. For any d-ple of these bundles, consider the set of 
colorings of the tree of depth n + 1 such that the first subtree of depth n is colored with a coloring from the 
first bundle, the second subtree with a coloring from the second bundle, etc. The resulting set of colorings 
on the depth n + 1 tree, has the following properties: (1) all colorings have the same colors at the leafs, and 
(2) the number (or fraction) of colorings in this set with a specific color at the root can be computed exactly 
using only the types of the bundles in the d-p\e. The resulting set of colorings on the depth n + l tree can 
be split again into bundles of the allowed types. The specific recipe for splitting this set into bundles of 
course will influence to what extent the balanced and near-balanced bundles dominate the entire collection 
of bundles. 

In the special case that the bundles are defined to be the set of colorings with a given boundary, i.e. all 
types of bundles are allowed and no splitting of bundles occurs, this gives exact recursive computation of 
the distribution of the marginal distribution at the root. 

In this next section we formalize and generalize the above construction. 
3.1 Definitions 

Let V denote a real vector space of finite dimension. Let S = (Si, ■ ■ ■ , S n ) be a finite sequence with Si € V. 
We denote the convex hull of S in V by (S). Let oti, . . . , ct n be functions from the convex hull (5) to [0, 1] 
such that the following properties hold for every 77 £ (S): 
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2 - »7 = Er=i«i( r ?) 

Thus for every r] G (5) these functions define a convex decomposition of 17. We will call the tuple 
(5, ai, . . . , a n ) a skeleton in V, and 5 the tese se? of the skeleton. 

By P we denote a distribution on V with finite support as well as a random vector chosen from this distri- 
bution. Let the skeleton (S, a±, . . . , a n ) be such that the support of the distribution of P lies inside (5). 
Let C be a random element of S with the following distribution: P(C = Si) = E[c*j(P)]. This is well 
defined by the first condition on the functions ai, . . . , a. n . We will call C a survey of P on the skeleton 
(S,a\, . . . ,a n ). 

We say that A is a survey of B without specifying the skeleton, whenever there exists a skeleton with respect 
to which A is the survey of B. 



3.2 Properties of surveys 



In this section we show several useful properties of surveys. 



Theorem 5 If P is a distribution on V with finite support, C is a survey ofP, and D is a survey ofC, then 
D is a survey of P. 



Proof: Suppose C is a survey of P on a skeleton (S, ai, . . . , a n ), and D is a survey of C on a skeleton 

(T, . . . , (3 m ). Let 7^(7?) := YTj=i a j(v)Pi{Sj) for i = l,...,n. Then (T, 71, . . . , j m ) is a valid 
skeleton, because for every 77 € V 

m m n n m n 

i=l i=l j'=l j=l i=l 3=1 

m m n n n n 

J2iMTi -EE'^^f^ = E a ^) E = E a ^)^ = ^ 

i=l i=l j=l j=l i=l j=l 

Finally, we can verify that D is a survey of P on the skeleton (T, 71, . . . , j m ). 

n n 

P(D = T,) = E^(C)] = ^P(C = ft^) = ^E[ aj (P)] ft(5 ; -) 

3=1 3=1 



E 



3=1 



E[7i(P)] 



Theorem 6 Le? Pi £OT<i P2 be independent distributions on V with finite support, and let Ci and C2 be 
their surveys. Suppose a distribution P on V is defined to be equal to Pi with probability p > and P2 
with probability 1—p, and similarly C is defined to be Ci w/f/z probability p and C2 vv/f/i probability 1—p. 
Then C c? survey of P. 
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Proof: Suppose Ci and C 2 are surveys respectively of Pi and P2 on skeletons (<S, a±,... ,a n ), and 
(T,Pi, . . . ,(3 m ). Let 1Z = S n T and \1Z\ = k. Without loss of generality, let's suppose that TZ = 
{Si, . . . , Sk} and Si = Ti for i = 1, r. Let's denote the union of the two basis sets as U = (U\, . . . , U m+n -k) 
(Si, . . . , S n ,T r+ i, . . . , T m ). This set will be the basis of the skeleton. Next we define the functions: 



'p aj( V ) P(Pi = 77) + (1 -p) fy(v) P(P 2 =V) j = l,-..,r 
P«j(l) P(Pi = v) 3 =r + l,...,n 

k (l -p) /3j-(n- r )(rf) P(P2 = v) j = n + l,...,m + n-k 



It is straightforward to check that (U, 71, ... , ^m+n-k) i s a valid skeleton. To check that C is a survey of P 
on this skeleton we just need P(C = Uj) = E[7j(P)]. For j = 1, . . . , r 

F(C = Uj) = p¥(C 1 = U j ) + (l-p)F(C 2 = U j ) 
= pE[a j (P 1 )] + (l-p)E[P j (P 2 )) 

= (P P ( p i = ? ?) + (1 - P) F ( p 2 = »7) ^W) 

= ^P(P = 7 ? ) 7 ,-(r?) 
= E[7i(P)]- 

Similarly, for j = r + 1, . . . , n, and for j = n + 1, . . . , n + m — we have respectively 

P(C = Uj)= p¥(C 1 = U j ) pE[«i(Pi)] =E[ 7 i(P)], 

P(C = I7j) = (1 -p) P(C 2 = ^-_ (n _ r) ) = (1 -p) E[/?,_ (n _ r) (P 2 )] = E[ 7i (P)], 



The next theorem is the one that allows us to use surveys in the context of the reconstruction recursion 
of Theorem 0] Let V\ , ■ ■ ■ , V r denote real vector spaces of finite dimensions. We say that a function / : 

Vi x • • • x V r . — > V is multi-affine if for every rji 6 Vi, 772 E V 2 , • • • , ??r E V r , a > and 6 > 0, such that 
a + b = 1, i G {1, . . . , r}, and 77^ G Vi, it holds that 

/(771, . . . , oT/i + 6r/-, . . . ,rj r ) = af(rj!, . . . ,rn, . . . , r] r ) + 6/(771, . . . , 77-, . . . , rj r ). 

Recall also that for a vector 77 G V we denote by \\r]\\ the sum of the coordinates of 77. 



Theorem 7 Let f : Vi x • • • x V r — > V be a multi-affine Junction. Let Pi, . . . ,P r be independent distribu- 
tions on Vi, . . . , V r with finite support, and Ci, . . . , C r be their respective surveys. IfQ and D are random 
elements ofV defined in the following way: 

P(Q = 77)ocE[||/(Pl,...,P r )|| Xlnd[/(Pl,...,P r )0C77]], 

P(D = ??) oc E [||/(Ci, . . . , C r )|| x Ind [/(Ci, . . . , C r ) oc 77]] , 
then D is a survey of Q. 
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Proof: We first demonstrate the proof for the case r = 1. We denote Pi by P and Ci by C. 



Suppose C is a survey of P on the skeleton (S,a±, . . . ,a n ). Consider the set {f(Si)/\\f(Si)\\ : 5, € 
S}. The size of this set can be less than n if for two different indexes 1 < i < i' < n it happens that 
f (Si)/\\f{Si)\\ = /(SV)/||/(SV)||. Let's denote by T = (T u . . . ,T m ) the ordered list of distinct elements 
corresponding to the above set. Let Ij = {i : f(Si)/\\f(Si)\\ = Tj}. 

Since the support of C is S it follows that the support of D is contained in T. It suffices to find 0i , . . . , j3 m 
non-negative functions on V such that P(D = Tj) = E[/3j(Q)]. We begin with the left-hand side: 



fD 



•x 



£>(C = Si) x x Ud[f(Si)/\\f(Si)\\ = Tj 



i=l 



^P(C = 5,) x 



^E[a t (P)]x ||/(5, 



ieij 



E 



iei-i 



Thus the normalization constant for the above probability is: 



E 



E 



E«*( p )^ 



a=l 



E[||/(P)II], 



where the second equality follows from the fact that / is multi-affine. 

Next, we look at the right hand-side of the desired equality. For every 77 G V we define 

W(j}) = E[||/(P)|| xInd[/(P)ocr7]], 
W = E[||/(P)||]. 



Then by the definition of Q, P(Q = 77) = W(rj) /W. For every j G {1, . . . , m}, let's define 



W(rf) 



E 



£>i(P) x 11/(^)11 x Ind[/(P) oc 77] 
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It is easy to check that Y1T=1 Pjiv) = 1> an d Sjli A^ 7 ?) = V f° r every t|£V. Finally, 



»;Gsupp(Q) 



W(»j) 



E 



J>i(P)x ||/(^)|| xInd[/(P)«i7] 



E 

r)Ssupp(Q) 



w ly(r?) 



E 



ieij 



£>i(P)x 11/(^)11 xInd[/(P)cx t?] 

= p(D = r i ) 



Next we generalize the proof to r > 1. Suppose Ci, . . . , C r are surveys on skeletons (S l , a\, . . . , a* ), 
. . . , (S r , a^, . . . , ). The same proof applies by defining the base set for the skeleton to be 



{f(SL...,Sl)/\\f(Sl,...,S r ir )\\ : Sj € 5 1 , . . . , S\ G S r } 



and changing the notation to Ij = {(h,. . . , i r ) : . . , S£.)/||/(S£, . . . , S[ 

(Pi,...,P r ). We have that 



Tj}, and P 



E 



[D = TV 



E[||/(P) 



E[0i(Q)], 



where 



E I14( p k) * 11/(4. X Ind[/(P) oc 7,] 

(h,...,i r )elj \k=l / 



Theorem 8 Let g be a convex function on V. For every distribution P onV of finite support, and C a survey 
ofP, E[g(P)] < E[g(C)] and the equality holds if g is an affine function. 



Proof: 



ng(C)] = E P [ C = Si]g(S i ) = ^2K[a i (P)]g(S i ) 



i=l 

E 



E«*( p )^) 



i=l 



> E 



g[J2<Xi(P)Si 



%(P)] 
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3.3 Using surveys to bound the probability of reconstruction 

Using the theorems from the previous section now we can show how to calculate recursively a survey of 
the residual distribution at the root of a random MRF on a skeleton of bounded size. Assume first that the 
degree distributions and potential function distributions for all levels have small support. Suppose we have 
a survey C for the random MRF of depth n. For level n + 1 first we calculate, for each possible instantiation 
of the degree r and potential functions . . . , ty r at the root, a survey of the residual distribution at the root 
using r copies of C and the recursion of Theorem @] The result is a survey of the true residual distribution 
for this instantiation by Theorem|7J Suppose the probability of this instantiation is p(r, . . . , \P r ). Next, 
we combine these distributions by defining a distribution equal to the surveys of each of the instantiations 
with probability p(r, ^i, . . . , ^ r ). This is a survey of the true residual distribution for the random MRF of 
depth n + 1 by Theorem [6l Finally, if the support of the resulting survey is bigger than the required bound, 
we can choose a smaller skeleton and compute a survey of the survey, which by Theorem [5] is also a survey 
of the true residual distribution. 

Finally, since distance from a fixed distribution is a convex function, by Theorem [8] the expected distance 
of the survey of the residual distribution from 7r is an upper bound on the distance of the true residual 
distribution from tt. The quality of this bound of course depends on how the skeletons were chosen at each 
step. 

If the bound on the skeleton size is b, the maximum degree possible for the tree is A, and the support of 
the potential-function distribution for every level is at most k, then the complexity of the computation of the 
survey is 0(n(kb) A ). The exponential complexity in A is in practice prohibitive, because in order to obtain 
surveys that give good bounds for the probability of reconstruction, b may have to be large. However, the 
important improvement here is that while the exact computation is exponential in n, computing the surveys 
takes time linear in n. 

A few more remarks regarding implementation are in order: 

1. It is not impossible to handle the case when the degree distribution has infinite support. The cases 
of the small degrees can be computed as above, and for large degrees, if the residual distribution is 
known to be symmetric (for example if the potential functions are symmetric), then one can use the 
trivial survey, the one whose basis set is the set of basis vectors. The uniform distribution on the basis 
vectors is a survey of every symmetric distribution. 

2. The size of the domains of the variables also influences the complexity of the algorithm. The compu- 
tation of the function / in the recursive step in general requires time \D Vl | x • • • x \D Vr \. However 
the more important factor is that the size of the skeleton may have to grow significantly with the size 
of the domain in order to obtain the desired bound. This dependence will have to be studied in the 
context of particular models. 

3. The strategy of choosing the skeleton at every step crucially influences the quality of the bounds. It 
may be that one type of skeleton is beneficial in the beginning iterations and a different type in the 
later ones. In our application we used small base sets in the beginning, which makes the first iterations 
faster, and refined the base sets (increased their size) progressively. Perhaps strategies for choosing 
the skeleton can be designed based on the current distribution (such as sampling a few probability 
vectors from it), but we have not found such a general-purpose strategy that performs well. 
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4. In order to obtain rigorous results, the computation of the survey of the residual distribution has to be 
earned out with rational numbers. However, naturally the size of the denominators increases expo- 
nentially, thus a "rounding" step is required at every iteration. In the case of symmetric distributions, 
such as those generated in the Potts model, this can be handled in a similar way to itemQ] We choose 
a large N which will be a bound on the size of the denominators. At every step the weights of all the 
vectors in the survey are rounded down to the nearest allowed rational number, one with denominator 
N, and the remaining weight is distributed among the basis vectors. The resulting distribution is a 
survey of the original one. By the symmetry, in the resulting distribution all the basis vectors have the 
same weight, therefore their denominator is at most \D V \N. 



4 Application to the Potts model on small-degree trees 

Let T be a random infinite tree rooted at the vertex p such that the number of children d of every vertex is 
distributed according to a random variable d with expected value d and maximum possible value d max . Let 
the domain of values that can be assigned to each vertex be denoted by D = {1, • • • , q}, and we will also 
call these values colors. The channel M on each edge in the Potts model is given by 

l-p if i = j, 
— ^— r otherwise, 

q-l ' 

where < p < 1. This channel corresponds to the g-state Potts model on the tree. Denote the resulting 
configurations of the tree by a and the alphabet at a vertex v by a v . The Potts model weighs the resulting 
configurations according to the Hamiltonian function H(a) = £V v ) eE (?<) Ind[er u = a v ] which counts 
the number of edges in which the color on both end points is the same. On a finite tree, the probability 
distribution is given by 

F(a) = | exp I (3 Ind[a u = a v ] 

\ (u,v)eE(T) 

where Z is a normalizing constant and f3 is an inverse temperature parameter of the Potts model. 
The second largest eigenvalue of the matrix M is denoted by 

A = l - pq = eP ~ l 
q-l + q-l 

In line with the terminology for the Potts model, A < corresponds to the ferromagnetic regime while A > 
corresponds to the anti-ferromagnetic regime. The special case of proper colorings corresponds to A = 

The branching number of an infinite tree is the supremum of the real numbers 7 > 1 such that T admits a 
positive flow from the root to infinity, where on every edge e, the flow is bounded by j~ £e , where £ e denotes 
the number of edges, including e on the path from e to the root. Note that for the d-ary tree, the branching 
number is d. In the case where each vertex of the tree has k children with probability p^, it is known that if 
the expected number of children m = ^2 k kpk > 1, then the branching number is m almost surely (see Q). 
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The Kesten-Stigum bound Q for the reconstruction problem says that for a tree with branching number 
d such that \d 2 > 1 reconstruction holds. For the Potts model, Mossel and Peres lfT2ll have shown that 
non-reconstruction holds if 



d 



qX 2 



< 1 



2 + \{q-2) 



and this bound was improved in (H. 

At q = 3, for large enough degree, the Kesten-Stigum bound was recently shown to be sharp in both the 
feiTomagnetic and antiferromagnetic cases |[T6l . However, there is still a gap for small degrees. We consider 
the anti-ferromagnetic and the ferromagnetic Potts models q = 3, on 2- and 3-ary trees, and on the tree 
in which every vertex is chosen with equal probability to be 2 or 3, and show bounds on the threshold for 
non-reconstruction. 

Let a denote a random configuration of a tree T given by the transition matrix. Recall that for a G D, L a (n) 
denotes the random coloring of L(n) conditioned on a p = a. In agreement with the notation of lfl6l we 
define the random variable 



This is the conditional probabilities of the color a at the root when the coloring of the vertices at level n is 
chosen conditioned on a p = a. Note that by the symmetry of the channel, the distribution of X + (n) does 
not depend on the particular a G D. 

Let Y + (n) := X + (n) - - and denote 



Here, the expectation is taken over the randomness of the tree and the Markov process on the tree (the 
random coloring). We go back to the unconditional distribution using the following identity of Sly ifToll : 

Claim 9 ( lfl6l ) The following relations hold: 



X + (n)=F L ^ La{n) (a p 



a\L). 



x n = E[Y + (n)], and z n = E[Y + (n) 2 }. 




Therefore, clearly, the condition 



lim x n = 



n 



oo 



is equivalent to non-reconstruction and further, if for each a, 




< e 



then x n < qe. 
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By expanding the recursion for the expectation E[Y + (n)] using a Taylor expansion, exactly as in |[T6l . 
we obtain a bound on x n+ \ in terms of x n and z n . Extending the analysis of lTT6l to random trees is 
straightforward. First, we obtain a relation in terms of a fixed degree d and then take the expectation over 
the distribution for the degree d. 



Theorem 10 Let q,d,d, d max , A, x n and z n be defined as above. Then, 



c n+ i < d\ 2 x n + E 
3=2 





x 2j 







2(g-l) fq(q-3 + X) q 2 X 



q-1 



q-1 



(g-l)(g-2) ( g(3g-6 + 2A) 

"T r, I / ^ » / „ N X n + 



2(9-1) 



2q 2 \ 



(q-l)(q-2) n (g-l)( g -2)' 
3~ 



9-1 



(6) 

(V) 
(8) 



Proof: The degree of the root of the random tree of depth n + 1 is d. We first bound the expectation 
conditional on d = d. Following the calculations of f!6l . we obtain 



E[Y+(n + l)|d = d] < 



2(9-1) 



A 2 g(g-3 + A) 
1 H : x r 

a - 1 



g 2 A 3 
9-1' 



(5 - 1)(9 - 2) / X 2 q(3q - 6 + 2A) 



9 Z 

2(9-1) 



2g 2 A 3 



A 



(9-l)(9-2) 
x n \ +l-l/q 



(9-l)(9-2)' 



9-1 



3=2 v/ 



2{q-l) fq(q- 3 + A) <? 2 A 



+ 



(g-l)(g-2) / g(3g-6 + 2A) 



(9-1) !) X " 4 
3~ 



2g 2 A 



2(9-1)/ 9 



9-1 



Taking the expectation over the degree we obtain the statement of the theorem. 

■ 

We obtain the following Corollary by applying Theorem[lO]to particular values of q and degree distributions 
d. The inequalities are obtained by optimizing each term in the summation © separately subject to the 
constraint that < z n < x n . 



Corollary 11 Let q = 3. In the ferromagnetic regime, 
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1. Ifd = 2 with probability 1, and A > 

x n +i < 2X 2 x n + h A x 2 n ,(2A 2 + 4A + 1) 

2. Ifd = 3 with probability 1, and A > 

Xn+i < 3A 2 x n , + h 4 x 2 n ,(2A 2 + 4A + 1) - ^A 6 x 3 (1 + A) 3 
5. Ifd = 2 with probability 1/2, d = 3 with probability 1/2, and A > 

xn+i < \\ 2 x n + 3A 4 x 2 (2A 2 + 4A + 1) - ^X 6 x 3 n (1 + A) 3 

In the antiferromagnetic regime, 

4. Ifd = 3 with probability 1, and A = — ^ 

3 63 2 351 3 

Xn+1 < ^X n + —X n - —X n 

Using the above bounds, we show that non-reconstruction holds in the following cases. 
Theorem 12 Let q = 3. In the ferromagnetic regime, 

1. ifd = 2 with probability 1, non-reconstruction holds for A < 0.69; 

2. ifd = 3 with probability 1, reconstruction holds for A < 0.555; 

3. ifd = 2 with probability 1/2, d = 3 with probability 1/2, non-reconstruction holds for A < 0.61. 
In the antiferromagnetic regime, 

4. Ifd = 3 with probability 1, and A = — \ {the case of proper colorings), there is non-reconstruction. 

Proof: Let x* = x*(X, q, d) denote the upper bound on x n given by the algorithm when it is run with inputs 
A, q, d. The values obtained for x* by the algorithm were as follows: 

• If d = 2 w.p. 1, A = 0.69, x* = 0.02939... 

• If d = 3 w.p. 1, A = 0.555, x* = 0.04457... 

• If d = 2 or 3 each w.p. 1/2, A = 0.61, x* = 0.04057... 

• If d = 3 w. p. 1 and A = -1/2, x* = 0.00038.. 
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d 


KS 


MP L12J 


FKRJ 


Theorem LL2J 


2 


0.7071.. 


0.6666.. 


0.7018.. 


0.69 


3 


0.5773.. 


0.5302.. 


0.5731.. 


0.555 


2.5 


0.6324.. 


0.5873.. 


0.6278.. 


0.61 



Table 1: The Kesten-Stigum upper bound on the non-reconstruction threshold and the values of A up to 
which non-reconstruction has been shown in lfT2l . |41 and here, for q = 3 in the ferromagnetic regime 
(A > 0). 

The values x* are an upper bound on x n . It can be verified by substituting the A above into the corresponding 
inequalities in Corollary [TT]that in each case x n+ \ < Cx n where C is a constant smaller than 1. This implies 
non-reconstruction since in the limit x n goes to 0. ■ 

For each of these cases the algorithm was run using Maple 12 and with only integer computations, using 
the rounding procedure described in the previous section. The base sets of the skeletons were selected 
manually, refining them whenever the resulting bounds stop improving. The decomposition functions of 
the skeletons were selected to minimize the expected total variation distance between the true vector and its 
decomposition using the LP solver of Maple 12. Not more than 100 iterations were needed for every case 
to obtain the required bound. The implementation was run on a MacBook with a 2GHz Intel Core 2 Duo 
processor and 1GB of RAM. The limiting factor is that the last tens of iterations take hours because the 
skeleton size we choose towards the end is close to 100 (in the case of d=2, 200). It is reasonable to expect 
that with more computational power or time each of these bounds can be improved, although going beyond 
the bounds of Formentin and Kiilske, if they are not tight, may require significantly better resources. 

The values of A we obtain for non-reconstruction in the ferromagnetic regime are shown in Table [4] for a 
comparison with previous bounds from (6) [121 SI (the bounds of (H are not explicitly derived, so we have 
not included them). The second column is the Kesten-Stigum bound below which reconstruction is known 
to hold. In all cases we improve the bound of lfl2l . The anti-ferromagnetic case (3-coloring on the 3-ary 
tree) is also not implied by the bounds of |[T2l . 
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